Statistically Significant Postsample Forecasting Improvements: How Big an Improvement is Likely Necessary?

Available methods for testing the statistical significance of an improvement in postsample forecasting from one model over another are briefly reviewed. These methods are based on a preselected partitioning of the data into a sample period (to be used for model specification/estimation) and a postsample forecasting period, to be used only for model comparison/evaluation. Given that one expects to obtain a postsample mean square forecasting error (MSFE) improvement of, say, 20%, how large a postsample period will be necessary in order for this improvement to be statistically significant at the 5% level? If the postsample forecast errors made by each model are NIID and these errors are independent of one another, then the table of 5% critical points for the F distribution provides the answer to this question. But forecast errors are typically substantially crosscorrelated – even for optimal forecasts made from well-specified models – and serially correlated as well . Here this question is examined for the case of crosscorrelated and serially correlated forecast errors, numerically generated from gaussian and truncated gaussian distributions.